08. Experiment Sizing

Experiment Sizing

Now that we have our main metrics selected: number of cookies as an invariant
metric, and the download rate and license purchase rate (relative to number of
cookies) as evaluation metrics, we should take a look at the feasibility of the
experiment in terms of the amount of time it will take to run. We can use
historical data as a baseline to see what it might take to detect our desired
levels of change.

Recent history shows that there are about 3250 unique visitors per day, with
slightly more visitors on Friday through Monday, than the rest of the week.
There are about 520 software downloads per day (a .16 rate) and about 65
licenses purchased each day (a .02 rate). In an ideal case, both the download
rate and license purchase rate should increase with the new homepage; a
statistically significant negative change should be a sign to not deploy the
homepage change. However, if only one of our metrics shows a statistically
significant positive change we should be happy enough to deploy the new
homepage.

Use the above information to answer the questions below.